{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.1 The problem of overfitting\n",
    "\n",
    "![](../pic/QQ截图20201023194050.png)\n",
    "\n",
    "![](../pic/QQ截图20201023194630.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.2 Cost function\n",
    "\n",
    "* regularization\n",
    "\n",
    "![](../pic/QQ截图20201023195613.png)\n",
    "\n",
    "![](../pic/QQ截图20201023195801.png)\n",
    "\n",
    "* Notation:（加入正则化项）\n",
    "\n",
    "$$\n",
    "J(\\theta)=\\frac{1}{2m}\\left[\\sum_{i=1}^{m}(h_{\\theta}(x^{(i)})-y^{(i)})^2 + \\lambda \\sum_{j=1}^n \\theta_j^2 \\right]\n",
    "$$\n",
    "\n",
    "* 正则化项的目标：尽可能地保持参数尽量小:\n",
    "\n",
    "$$\n",
    "\\lambda \\sum_{j=1}^n \\theta_j^2\n",
    "$$\n",
    "\n",
    "* 如果$\\lambda$值太大，则损失函数中起主要作用的就是第二项（正则化项），即$\\theta_1=\\theta_2=...=\\theta_n = 0$，只剩下$\\theta_0$。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.3 regularized linear regression（线性回归正则化）\n",
    "\n",
    "* gredient descent\n",
    "\n",
    "$\n",
    "Repeat\\{\n",
    "$\n",
    "\n",
    "$\n",
    "\\quad \\theta_0 := \\theta_0 - \\alpha \\frac{1}{m}\\sum_{i=1}^{m}(h_{\\theta}(x^{(i)})-y^{(i)})x_0^{(i)}\\\\\n",
    "\\quad \\theta_j := \\theta_j - \\alpha \\left[\\frac{1}{m}\\sum_{i=1}^{m}(h_{\\theta}(x^{(i)})-y^{(i)})x_j^{(i)}\\right] + \\frac{\\lambda}{m}\\theta_j\n",
    "$\n",
    "\n",
    "$\n",
    "\\}\n",
    "$\n",
    "\n",
    "* note:\n",
    "\n",
    "$$\n",
    "\\quad \\theta_j := \\theta_j(1-\\alpha \\frac{\\lambda}{m}) - \\alpha \\frac{1}{m}\\sum_{i=1}^{m}(h_{\\theta}(x^{(i)})-y^{(i)})x_j^{(i)}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.4 regularized logistic regression（逻辑回归正则化）\n",
    "\n",
    "* Cost function for logistic regression\n",
    "\n",
    "$$\n",
    "J(\\theta)=\\frac{1}{m}\\sum_{i=1}^{m}Cost(h_{\\theta}(x^{(i)}),y^{(i)})\\\\\n",
    "=-\\frac{1}{m}[ \\sum_{i=1}^{m}y^{(i)} \\log (h_{\\theta}(x^{(i)})+(1-y^{(i)}) \\log(1-h_{\\theta}(x^{(i)}))]\\\\\n",
    "$$\n",
    "\n",
    "* add the regularized\n",
    "\n",
    "$$\n",
    "J(\\theta)=-\\frac{1}{m} \\left[ \\sum_{i=1}^{m}y^{(i)} \\log (h_{\\theta}(x^{(i)})+(1-y^{(i)}) \\log(1-h_{\\theta}(x^{(i)}))\\right] + \\frac{\\lambda}{2m}\\sum_{j=1}^{n}\\theta_j^2\\\\\n",
    "$$\n",
    "\n",
    "\n",
    "$\n",
    "Repeat\\{\n",
    "$\n",
    "\n",
    "$\n",
    "\\quad \\theta_0 := \\theta_0 - \\alpha \\frac{1}{m}\\sum_{i=1}^{m}(h_{\\theta}(x^{(i)})-y^{(i)})x_0^{(i)}\\\\\n",
    "\\quad \\theta_j := \\theta_j - \\alpha \\left[\\frac{1}{m}\\sum_{i=1}^{m}(h_{\\theta}(x^{(i)})-y^{(i)})x_j^{(i)}\\right] + \\frac{\\lambda}{m}\\theta_j\n",
    "$\n",
    "\n",
    "$\n",
    "\\}\n",
    "$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
